Wide dynamic range image capturing system method and apparatus

ABSTRACT

An image capture system is presented where the dynamic range of photo imaging devices, such as a still or video camera, is increased by varying sensor exposure time on a pixel-by-pixel basis under digital camera processor control. The systems photo sensors are continuously illuminated without reset over the exposure interval. In addition to being interrogated at the end of the exposure interval, the pixels are also non-destructively interrogated at one or more intermediate times during the interval. At each interrogation, the image capture system determines individually whether the pixels have saturated and if not, the parameter value is recorded; if the pixel has saturated, the previously stored value from the preceding interval is maintained. To produce the final sensor value for the whole exposure interval, the data for pixels that reached the saturation level are adjusted to compensate for their shortened exposure.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.11/467,993, filed on Aug. 29, 2006, which is hereby fully incorporatedby reference.

BACKGROUND OF THE INVENTION

The present invention relates generally to image capturing systems, and,more particularly, to increasing the dynamic range of image sensingdevices, such a CMOS pixel circuits, having limited dynamic range.

Digital cameras and other electronic image capturing systems have becomeincreasingly common. In such systems, photographic film is replaced byan array of photo sensors. Although such systems have a number ofadvantages with respect to film based system, they typically lack thedynamic range of photographic film.

A common type of image sensing element uses CMOS active pictureelements. Such image sensor devices of the CMOS active picture elementvariety rely on having an image sensing area divided into a large numberof photosensitive areas, or pixels. These areas produce an output signaldetermined by the light incident on the physical location of the pixel,with the pixel information being read out to a set of externalelectronics following a predetermined scanning sequence. Randomaddressing, sequential addressing, or some combination of the two may beused. A typical pixel circuit using three transistors is shown in FIG.1.

In FIG. 1, the diode D1 is normally biased in its reverse direction, sothat if it is previously charged to a voltage, photons impinging on thediode will produce carriers which discharge its terminal voltage as seenat wire 3. Transistor M1 serves as a reset transistor for recharging thediode voltage. When the row voltage VRES on wire 1 is taken in asufficiently positive direction, it causes conduction in transistor M1and charges the diode voltage towards the voltage VDD on wire 4. Thisreset action is normally initiated at the start of a time when the diodewill be used to accumulate photon information about the impinginglight's intensity.

After a predetermined exposure time, the light intensity information forthis pixel is read out by using the transistors M2 and M3. Transistor M2serves as a source follower, with the voltage on its source at 5 being afunction of the diode voltage at 3. The voltage source VDD on wire 4provides a current to operate the source follower. When the particularrow of pixels containing this pixel is selected for readout, the rowvoltage VSEL on wire 2 is taken in a positive direction, turning ontransistor M3. Transistor M3 is usually operated as a switch, to connectthe source terminal of M2 to the readout column line 6. A current sink Iat the end of the column line provides operating current for the sourcefollower M2, so that the output voltage on the column line VOUT will beproportional to the voltage on the diode at 3.

After the intensity information is read out and is no longer needed, therow reset input VRES may be activated to cause the pixel voltages to berestored to the value representing zero light intensity. In addition tofunctioning as a reset transistor, M1 may also be used to limit thelower voltage to which wire 3 may descend. This prevents diode D1 frombecoming zero or forward biased, and therefore prevents the injection ofminority charges into the substrate. Charges injected into the substratemay diffuse to adjacent pixels, causing spreading or blooming in theimage of bright image spots. In many systems, this anti-blooming featureis used to hold the pixel voltages at their lower level until the resetaction is initiated to prepare for a new image exposure.

For simple image sensor usage, the pixel in FIG. 1 may be formed into anarray of cells with control and readout circuitry surrounding it on asingle silicon integrated circuit. FIG. 2 shows an exemplarytwo-dimensional image sensor conceptual diagram.

In the two-dimensional imager of FIG. 2, a rectangular area 10 ispopulated with an array of pixels such as those shown in FIG. 1, withsome of the row and column wires shown for clarity. Control logic 11,primarily located at an edge of the pixel array, operates the VSEL andVRES wires for each row of the pixel array. Counters, gates, and/orshift registers in this logic generate the control signals needed tofollow the desired pixel readout sequence. When a pixel row is selected,the information from it goes to the readout circuitry 12, as signals onthe columns wires 13. The resulting pixel information is output on 14 aseither analog or digital signals as needed.

For normal sequential scanning, the control logic 11 activates one rowreset signal VRES at a time, following a linear scan pattern descendingfrom the top of the pixel array 10 to the bottom. Consider for thisexample that at the time of interest, the VRES signal is being appliedto row A. At the same time, the control logic also sends the selectsignal VSEL to a different row, which we may choose to be row B in thisexample. The pixel information from row B will then be sent to thereadout circuitry 12 on the entire set of column wires, of which 13 isan example, and the readout circuitry will choose which columninformation to send out on the connection 14.

The total pixel exposure interval for a row of image sensor pixels isdetermined by the delay time between the application of a first VRES anda second VRES to a particular pixel row. During this time period thelight illuminating the image sensor is not interrupted. The amount ofsensor pixel row exposure per interrogation cycle interval is determinedby the delay time between the application of the VRES and the VSEL to aparticular sensor pixel row. If rows are being scanned in a regularperiodic manner, this time interval will be determined in turn by therow number spacing between row A and row B. Thus the amount of time theimage sensor is exposed to light will be a function of the row timingdelay between the row reset action and the row readout action asdetermined by the control logic. Image readout from a different row,such as row C with a different relative location, will give differenteffective image exposure time.

Although this discussion is based on one of the simpler three transistorcells in use, this configuration exhibits the sort of limited dynamicrange due to saturation that is also found in the various other imagesensing element designs known in the art. Compared to the dynamic rangeavailable from film, such elements will consequently exhibit saturationwhen exposed to a bright image while still being under exposed for adark image.

There are numerous methods and apparatus that have been proposed toincrease the dynamic range of photo sensors in general, and CMOS sensorsin particular. Perhaps the most straightforward is just to increaseamount of charge that is stored by the diode D1 of FIG. 1; however, thisresults in a larger pixel circuit, when the trend is to reduce pixelsize in order to increase resolution.

The prior art present a number of alternate techniques for dealing withthese problems; however, these suffer from one or more serious practicalproblems, such as requiring non-standard sensors, increased memoryrequirements, or multiple sensor resets. For example, U.S. Pat. No.6,977,685 is illustrative of the class of CMOS sensor dynamic rangeenhancement technologies that rely upon resetting individual pixelsduring the course of photo sensor exposure to light as the primary meansof effecting dynamic range extension. In U.S. Pat. No. 6,946,635 thesensor pixels are read out two or more times without resetting the imagesensor, but require that these readings are separated by a shutteropening and closing cycle at the point of sensor (photo detector)re-exposure, so that the exposure of the array is not continuous for asingle image. U.S. Pat. No. 6,115,065 employs a specific, specializedimage sensor that incorporates a memory cell with each image sensorpixel and calls for resetting and reading each sensor pixel site twice,one over a short exposure interval and one over a long exposureinterval, and clocking out both of these values from the photo sensor.According to the teachings of U.S. Pat. No. 6,018,365, the dynamic rangeof an imaging system that utilizes an array of active pixel sensor cellsis increased by reading each cell in the array multiple times duringeach integration period, saving the number of photons collected by thecell, resetting the cell if it would normally saturate by the end of theintegration period, and summing of the values collected during theintegration period, an approach requiring both the resetting of cellsand additional memory. United States Patent Application 20020067415offers a method of operating a solid state image sensor having an imagesensing array that includes a plurality of active pixels, the resettingof each pixel, and after successive time periods, reading outputs fromeach pixel, to obtain multiple sets of image data having differentdynamic ranges that are then combined, an approach again requiring boththe resetting of cells and additional memory.

All these various approaches suffer from the need to either usenon-standard CMOS sensors that are significantly more costly tomanufacture and control than the CMOS sensors commonly in use, orrequire significantly more image buffer memory to properly operate. Oneproposed approach for increasing CMOS sensor dynamic range, while usingstandard CMOS sensors, is to capture two to more images, taken veryclose together in time, at different exposures, at least one whichdisplays the shadow detail of the image (taken at a high exposure) andat least one which displays the highlight detail (taken at a lowexposure). These multiple images, which need to be corrected for anymotion present in the images occurring between individual image capture,are combined into a single image that displays both shadow and highlightdetail. This approach does result in an increase of dynamic range;however, the processing complexity to motion compensate andappropriately combine the two or more captured images, and theadditional buffer memory required, is excessive by current standards,and therefore this solution, to date, has not been shown as a practicalanswer to the problem. The need to use specialized, non-standardsensors, add memory, or reset pixels during the expose interval all haveserious drawbacks. Consequently, there is a need for improved techniquesto increase the dynamic range of photo sensors.

BRIEF SUMMARY OF THE INVENTION

The present invention relates to extending the dynamic range of photoimaging devices, such as a still or video camera, by varying sensorexposure time on a pixel-by-pixel basis under digital camera processorcontrol. This can be achieved using a largely standard CMOS sensor thatis continuously illuminated, and sensitive to illumination, during theentire exposure interval. Thus, present invention represents apractical, cost effective approach acceptable to CMOS sensormanufacturers to significantly expand the dynamic range of imagingsystems, such as digital cameras, that use CMOS sensors.

The present invention illuminates the photo sensors continuously withoutreset over the exposure interval. Once a pixel is reset, in addition tobeing interrogated at the end of the exposure interval, it is alsonon-destructively interrogated at one or more intermediate times duringthe interval. For instance, in one exemplary embodiment, the interval isdivided into ten equal subintervals. Other embodiments may use adifferent number of subintervals, unequal intervals, or both. At eachinterrogation, the device determines individually whether the pixelshave saturated and if not, the parameter indicative of the cumulativeexposure level (such as the voltage in a CMOS sensor) is checked to seeif has reached a limit value (such as the saturation level): if thepixel has not saturated, the parameter value is recorded; if the pixelhas saturated, the previously stored value from the preceding intervalis maintained. To produce the final sensor value for the whole exposureinterval, the data for pixels that reached the saturation level areadjusted to compensate for their shortened exposure. This adjustment canbe effected by keeping track of which exposure sub-interval the recordedpixel data corresponds to, and scaling the recorded pixel accordingly,based on the relative duration of the subinterval to the full interval.Such adjustment can be performed (a) after the full exposure interval isover and all the pixel data collected or (b) “on the fly”. If performed“on the fly” those pixels that have not reached the limit value can beadjusted immediately following the last interrogation cycle. For thosethat reach the limit value, the adjustment is performed immediatelyfollowing the reaching of such limit. In either case, only one datavalue for each pixel is needed.

The individual photosensitive pixel elements can be formed into an arrayto produce an imaging system. As the rows of the array are reset, fromtop to bottom for example, the exposure interval begins for each row ofpixels. Once the last row is reached, the first row is returned to andthe rows of the array are non-destructively interrogated at anintermediate time prior to the end of the exposure interval. Thisnondestructive read during the exposure interval can be performed atmultiple intermediate times until, at the end of the exposure interval,the integration of pixel data, corresponding to the full exposureinterval, is made. At the conclusion of each of the interrogationcycles, a processor makes computations on the received pixel data, wherethese computations are responsive to image pixel data acquired fromprevious interrogations, in order to provide image pixel data withimproved dynamic range. In a variation of this approach, the rows of thearray are split into subgroups that are scanned separately, allowingmore time for the individual scans.

As the present invention can introduce multiple effective exposure timeswithin a single image, in addition to expanding the usable dynamic rangeof an imaging system, it can be used to provide a sharper image toportions of the frame. As brighter areas of the image will use pixeldata for only a portion of the full interval, they will correspondinglydisplay less motion artifacts. In alternate embodiments, the use ofvariable exposure times within the single image can be user selected.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 shows an example of a three-transistor pixel circuit;

FIG. 2 illustrates a two-dimensional imager;

FIG. 3 shows the cumulative light incident on a pixel over an exposureperiod for several intensities;

FIG. 4 is a flow chart of an exemplary embodiment as seen by a pixel;

FIG. 5 shows a portion of an image selected for manipulation; and

FIG. 6 is a block diagram of a wide dynamic range image capturingsystem.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

As described in the Background section, for CMOS and other sensorscommonly used in today's commercially available digital cameras, theexposure time per pixel is the same for each pixel in the array. Thispredefines the dynamic range of the CMOS sensor for a given picture. Inhigh contrast scenes, there are always conflicts between providingsufficient exposure time to capture details in the dark, shadow regionsof an image, while simultaneously preventing saturation of bright,highlight regions of the image. The present invention discloses themeans to vary sensor exposure time, on a pixel-by-pixel basis, underdigital camera processor control, while using an almost standard CMOSsensor that is continuously illuminated, and sensitive to illumination,during the entire exposure interval. Thus, the present inventionrepresents a practical, cost effective, approach, acceptable to CMOSsensor manufacturers, to significantly expand the dynamic range ofimaging systems, such as digital cameras, that use CMOS sensors. Inaddition to CMOS sensors, other types of image sensors can be used withthe present invention. The characteristics these sensors should possessare that (1) they can be non-destructively interrogated, multiple times,during a single, continuous exposure interval and (2) they built-up, ordeplete, a signal level that is responsive to the cumulative amount oflight falling on a photosensitive element of the sensor during theinterval, such as the charge on a storage element (the photo diode inthe case of CMOS sensors) during the exposure interval.

According to its various aspects, the present invention improves imagequality of an electronically captured image, by preserving image detailsin the highlights and shadows of the captured image. Therefore, capturedimages will look more like they were captured on photographic film.Further, unlike many prior art image dynamic range extension schemes,such as described above in the Background, the approach of the presentinvention is scalable with image sensor size, without the need to addcostly additional image buffer memory. Consequently, the presentinvention can be implemented with little increase in cost, bothinitially and in the future, as photo sensor pixel densities increase.

The situation can be illustrated with respect to FIG. 3. FIG. 3 showsthe cumulative amount of light falling on a sensor pixel versus timeover an exposure time T. Time t runs along the horizontal axis beginningat the start of an exposure interval and with the end of the exposureinterval at a time T. The vertical axis is the cumulative incidentlight. This is shown for three different light intensities, 101, 102,and 103, which can either be taken to correspond to three differentpixels during the same exposure or be taken to correspond to the samepixel for different exposures. For each of these lines, the lightintensity is taken to be constant over the interval, resulting in theshown straight lines, although the discussion of a more generalsituation readily follows.

The lowest light intensity 101 results in the amplitude A₁(T) at the endof the exposure period, while the higher light intensity level 102gives, or at least should give, the amplitude A₂(T). However, if thesensor measuring this has a saturation level, S, and the accumulatednumber of photons exceeds this level, any pixel whose incident lightexceeds this level will instead return the value S instead of the actualvalue. For example, rather than return the correct amplitude A₂(T) forlevel 102, the saturation level S is returned instead. To overcome thislimitation, according to a principle aspect of the present invention,the amplitude is measured at one or more intermediate times. When theamplitude is measured at the next intermediate time, or final time, thisprevious amplitude is replaced by the new amplitude only if the is belowthe saturation level. If the amplitude obtained at one of theintermediate times is used, it needs to be compensated, for it onlycorresponding to a portion of the entire exposure time.

For example, in FIG. 3 an intermediate time t′ is introduced. For theexposure level of line 101, the amplitude can be measured at time t′.The sensor continues to be exposed and when measured again at the end ofthe interval, as A₁(T)<S, the value A₁(T) replaces the value recorded att′ and is used.

For the exposure level of line 102, the amplitude measured at time t′ isA₂(t′). The sensor again continues to be exposed, but reaches thesaturation level S before T, which result in the saturation value S,rather than A₂(T), being recorded for the pixel, if the value A₂(t′) isoverwritten. Since A₂(t′) is the amount of signal accumulated inresponse to the light falling on the sensor pixel, up to exposure timet′, and the exposure time t′ is less than the entire exposure interval,A₂(t′) needs to be compensated as follows:A ₂(T)=(T/t′)*A ₂(t′).

For example, if t′=½ T, then A₂(T)=2*A₂(t′). Consequently, the dynamicrange is effectively extended, while only a single amplitude needs to bekept for each pixel, along with a number that indicates the fraction ofthe total exposure time that had elapsed at the time the pixel amplitudewas stored. Alternatively, only the compensated value need be kept, ifpixel dynamic range processing is performed “on the fly” at the time ofpixel acquisition.

If the light level were high enough, as at 103, the dynamic range can beextend further introducing more intermediate times, such as times t″ andt′″. In this example, at the level shown at 103 saturates between t″ andt′, the value A₃(t″) would be used and compensated accordingly. (Notethat it is the Last Value Before Saturation (and an Indication of whichTime it Corresponds) that is kept, and then compensated, and not thefirst saturated value. Thus, for 103 the value used isA₃(T″)=(T/t″)*A₃(t″), not A₃(T′)=(T/t′)*A₃(t′) The number of suchintermediate times introduced is a design choice that needs to balancethe amount of dynamic range increase desired against complexity and readout speed.

Although FIG. 3 shows the cumulative amount of incident light falling ona pixel during an exposure interval (which is monotonicallynondecreasing), what is typically happening in a given sensor, such asCMOS photo sensors like that described with respect to FIG. 1, is that aquantity (such as the amount of charge on the diode) is decreasing inresponse to incident light until it hits the saturation level.Consequently, the parameter that is actually measured in the device ofFIG. 1 is a voltage, V, of the form V=V_(reset)*−f(A), where V_(reset)is the reset value and f(A) is some increasing function of the totalcumulative exposure.

Returning to FIGS. 1 and 2, the description of the various aspects ofthe present invention can be described in more detail using thearrangement given there for some exemplary embodiments. As describe inthe Background, after a predetermined exposure time, the light intensityinformation for this pixel is read out by using the transistors M2 andM3, where transistor M2 serves as a source follower, the voltage on itssource at 5 being a function of the diode voltage at 3, which is turn anindication of the amount of light to which it has been exposed. Whenthis particular row of pixels is selected for readout, the row voltageVSEL on wire 2 is taken in a positive direction, turning on transistorM3, which is usually operated as a switch to connect the source terminalof M2 to the readout column line 6. A current sink I at the end of thecolumn line provides operating current for the source follower M2, sothat the output voltage on the column line VOUT will be proportional tothe voltage on the diode at 3. It is important to note that the processof reading the intensity information from the pixel diode D1 does notinject any net charge to wire 3, so the readout process may be repeatedwithout earlier readout actions affecting the later values. In otherwords the readout is “non-destructive”.

After the intensity information is read out and is no longer needed, therow reset input VRES may be activated to cause the pixel voltages to berestored to the value representing zero light intensity. In addition tofunctioning as a reset transistor, M1 may also be used to limit thelower voltage to which wire 3 may descend. This prevents diode D1 frombecoming zero or forward biased, and therefore prevents the injection ofminority charges into the substrate. Charges injected into the substratemay diffuse to adjacent pixels, causing spreading or blooming in theimage of bright image spots. In many systems, this anti-blooming featureis used to hold the pixel voltages at a lower level until the resetaction is initiated to prepare for a new image exposure.

The device of FIG. 1 is one of the simpler three transistor cells inuse, and various changes may be made to this design as known in the artto produce equivalent operation with additional advantages. For thepurposes of the invention disclosed here, the exact structure of thepixel cell is not important, as long as the pixel may be read out in anon-destructive manner, at various times of our choosing, after thepixel charge is reset to a standard value. The readout information maytake the form of either voltage or current, and may be read from acolumn line connected to either the source or the drain of the readouttransistor circuit without affecting the operation of main aspects ofthe present invention.

As is clear from the discussion given with respect to FIG. 3, therelevant property of a sensor is that, within some range, it has someparameter that is either decreasing or increasing as it is exposed tolight during an exposure period and that, either by choice or because itis inherent in the device, there is some limit to the range of thisparameter. For the exemplary device in FIG. 1, the parameter is theamount of charge on line 3, which decreases from its initial reset valueuntil either the exposure period ends or the saturation level isreached. If this parameter can be read non-destructively withoutresetting while exposure continues, the present invention can be used toextend the dynamic range of an imager made up of such devices. In theexemplary embodiments, the exposure period is broken up into tensubintervals of equal length, but, more generally, a different number ofintermediate interrogations and subintervals of differing relativedurations can be used.

In other embodiments, the exposure period can be broken up into a numberof unequal intervals. Such an arrange can be used, for example, when thesensor does not respond linearly and its sensitivity (such as theincremental voltage output per incremental photon input) decreases asthe sensor gets closer and closer to its saturation level. By increasingthe time between interrogation cycles at the end of the exposureinterval, this type of non-linearity can be compensated for.

A first embodiment of the present invention employs the simple photosensor of FIG. 1 and the pixel row and column arrangement of FIG. 2.Further, it employs the non-destructive interrogation of each row of thesensor pixels two or more times, for example, starting from the top ofthe sensor, and proceeding to the bottom of the sensor to significantlyincrease the effective dynamic range of the photo sensor. As discussedwith respect to FIG. 2 in the Background section, the amount of imagesensor pixel row exposure per interrogation cycle interval is a functionof the row timing delay between the row reset action and the row readoutaction, as determined by the control logic. In most cases, this logic isdriven by timing signals from the image capturing systems' (a digitalcamera for example) digital processor. Therefore, the amount of timeeach row has been exposed to light, at the instant the row isinterrogated, is known a priori by the digital camera's processingengine. With this in mind, the method and apparatus employed by a simpleexemplary implementation of the present invention could proceed throughthe following steps:

(1) Reset all rows of the image sensor starting from the top of thesensor and proceeding to the bottom of the sensor.

(2) Upon completion of resetting the last row of the image sensor, begininterrogating data from the first row of the sensor. Thus, exposure timefor this first sensor pixel row will be equal to the time employed toreset the sensor array.

(3) Interrogate all rows of the image sensor, from the first pixelsensor row to the last pixel sensor row, over a time period equal tothat employed to reset the entire image sensor array. This causes thefirst row to be reset to be the first row to be read, and the last rowto be reset to be the last row to be read. Since the reset cycle timeequals the interrogation cycle time in this example, each sensor pixelrow will be exposed to light for the same amount of time before beingread, assuming uniform row scanning. For example, if the reset andinterrogation cycle times are 3 milliseconds, each pixel sensor rowwould be exposed for 3 milliseconds before row readout is initiated forthat row.

(4) For this exemplary embodiment, a total of 10 interrogation cycleswill be executed per photo sensor exposure interval. After reading thelast pixel sensor row, the 1^(st) interrogation cycle will have beencompleted. Immediately thereafter, the 2^(nd) interrogation cycle isstarted, which is then followed by the 3^(th), 4^(th), 5^(th), etc.interrogation cycles, until the 10th interrogation cycle has beencompleted.

(5) Read each row of sensor pixel data from the 1^(st) interrogationcycle and store this pixel sensor row data in an image buffer memory.This data represents pixel sensor output produced at 1/10 total exposureinterval.

(6) Read each row of sensor pixel data from the 2^(nd) interrogationcycle. This data represents pixel sensor output produced at 2/10 totalexposure interval. Overwrite the sensor pixel data in the buffer memoryfrom interrogation cycle 1 with sensor pixel data from interrogationcycle 2 if the sensor pixel data from interrogation cycle 2 is below apredefined threshold. This threshold flags sensor pixel data that issaturated or close to saturation. When a sensor pixel location in memoryis not overwritten, because the sensor pixel data from interrogationcycle 2 is above the predefined threshold, store an indication in memorythat this image pixel was acquired from the 1st interrogation cycle.

-   -   Note that under this arrangement, no additional image buffer        memory is needed, as only one value per pixel is kept, but there        is a need to keep track of which interrogation cycle (1 to 10 in        this example) the recorded value corresponds to. Note also that        the exposure is continuous, without the sensors being reset, as        all of the reads are nondestructive.

(7) Read each row of sensor pixel data from the 3^(rd) interrogationcycle. This data represents pixel sensor output produced at 3/10 totalexposure interval. Overwrite the sensor pixel data in buffer memory frominterrogation cycle 2 with sensor pixel data from interrogation cycle 3,if the sensor pixel data from interrogation cycle 3 is below apredefined saturation threshold. When a sensor pixel location in memoryis not overwritten, because the sensor pixel data from interrogationcycle 3 is above the predefined saturation threshold, store anindication in memory that this image pixel was acquired from the 2^(nd)interrogation cycle.

(8) Continue the image sensor interrogation process outlined in (6)until all 10 interrogation cycles have been executed. After the 10^(th)interrogation cycle, the image buffer memory contains a complete imagecomposed of sensor pixel data captured at exposure times ranging from1/10 total pixel exposure interval to 10/10 total pixel exposureinterval. Along with the stored data for each image pixel is anindication of the interrogation cycle number from which it was acquired.Note that none of the pixel data stored in the image buffer memory arepositively saturated, that is, saturated due to excessive exposure tolight.

(9) By use of the digital camera's processor, multiply the value of eachsensor pixel by the total number of interrogation cycles divided by theinterrogation cycle number from which it was acquired. This processplaces the effective value of each pixel at the point in the dynamicrange of the output digital image that corresponds to the intensity ofthe pixel in the captured physical scene.

(10) An alternative to storing the value of each sensor pixel in theimage buffer memory, along with interrogation cycle number from which itwas acquired, is to overwrite the value of a stored pixel by its storedvalue multiplied by the total number of interrogation cycles divided bythe current interrogation cycle number −1, when it found that the pixelvalue for that location, as read during the current interrogation cycle,is above the predefined saturation threshold. This places the effectivevalue of each pixel in the image buffer at the point in the dynamicrange of the output digital image that corresponds to the intensity ofthe pixel in the captured physical scene without the need for:

-   -   a) additional memory to hold interrogation cycle number        information for each pixel in the image;    -   b) a subsequent operation on the stored image buffer data by the        digital camera's processor.

As described, it is possible that one or more pixels of a sensor maysaturate before the first interrogation is performed; however, when thisoccurs the camera (or other imaging system) can reduce the sensitivityof the image sensor using the overall exposure control mechanisms of thecamera. This can be done by reducing the size of the digital camera'saperture until all pixels of the sensor no longer saturate. Since therange of light reduction due to aperture closure can be very wide, fromfull light falling on the sensor to complete light blockage, thecircumstance of portions of the image sensor being saturated before thefirst read can be prevented. Note that under the arrangement of thepresent invention, the maximum aperture setting can be based on theexposure level at the end of the first exposure subinterval, rather thanthe full interval, allowing darker regions the benefit of longerexposures and the consequent better definition. In some cases, however,there is often a desire on the part of the camera user to emphasiscertain portions of the image at the expense of other portions of theimage, therefore this camera behavior can be overridden by the user, ifdesired, to obtain a particular image effect. This desire is mainlyassociated with the limited dynamic range of prior art approaches. Notethat the present invention significantly reduces the need for such userintervention because of the increased image dynamic range provided.

The above scenario called for 10 non-destructive reads of the imagesensor during a single pixel exposure interval. This results in a 10times increase in the dynamic range of the captured image. A greater orlesser number of non-destructive reads of the image sensor during asingle exposure interval would increase or decrease the dynamic range ofthe captured image proportionally. Note that negative pixel saturation,that is the loss of image details due to insufficient sensor exposure tolight, is prevented by lengthening the overall exposure time andnon-destructively reading the image sensor a greater number of times.Since the pixels from the bright parts of the image use a shortereffective exposure time, when this approach is used the bright areas ofthe image will be sharper and display less motion artifacts than the dimareas of the image.

FIG. 4 is a simplified flowchart of the above process as seen by a givenpixel, or row of pixels, beginning with the sensor being reset at 201and the exposure beginning at 203, where the index n introduced andinitialized to n=1 for the purposes of this flow at step 205 in order tokeep track of the exposure sub-intervals. As discussed above, in oneaspect of the present invention, the sensor is exposed continually,without reset, for the entirety of the exposure period. The parameterindicative of the amount of cumulative light to which the pixel isexposed will be called the amplitude A for this discussion and, forsimplicity, taken to increase with exposure. In the actual sensor, theparameter measured may actually decrease, as in the CMOS sensor of FIG.1 where a voltage V starts at a reset value, Vreset, and dischargesbased on light intensity.

The first interrogation, at time t₁, is made at 207 and the amplitudeA(t₁) of the pixel is recorded at 209, with the index n beingincremented at 211 as the process enters the loop for the remainingsubintervals.

At the end of the next subinterval, t_(n), the cumulative amplitudeA(t_(n)) is read at 213 and step 215 determines whether the limitvalues, such as the saturation level S, has been reached. If A(t_(n))<S,the pixel value is overwritten at 217. (It should again be noted thatwhat is actually being measured is some parameter indicative of thisexposure amplitude and that this parameter may actually decrease withexposure: for example, in the exemplary embodiment it is the voltage atthe diode which is measured and this parameter starts at the reset valueand decreases until either the exposure ends or it reaches thesaturation level. Consequently, the correct statement of step 215 inthis case would be V>V_(saturation).)

Should the limit be exceeded, an indication is then set at 219 to notethat the recorded amplitude corresponds to previous loop (n−1 at thispoint) and that this will be the value used to construct the finalvalue. It should be clear that a number of implementations can be usedfor this step. For example, as the amplitude for each value of n isrecorded, the corresponding value of n is also recorded and once thelimit is reached, neither n nor A(t_(n)) are updated further, so that atstep 219 the indicator (n value) has already been set in the precedingloop and is just not updated. At this point the pixel can either remainin the loop, with the yes path out of 215 being taken in each furtheriteration, or it can just be taken out of the loop and wait for the fullexposure period to end.

The flow then continues on to step 221 to determine whether the end ofthe full exposure interval has elapsed and the final interrogation hasbeen made: if not (n<N), the flow loops back to 211; if so (N>n), theexposure is ended (223). At this point, the amplitude written for eachpixel will be A(t_(N)) (or [A(t_(N)),N]) if the limit was not reachedand [A(t_(n)), n] if it did reach its limit, where n is the index of thelast sub-interval before the limit was reached. Step 225 thencompensates the amplitudes for the written pixel amplitude values, usingthe equation A(t_(N))=A(t_(n))*(n/N), so that a pixel amplitudeequivalent to the amplitude that would have been obtain over the fullexposure interval, should saturation not have occurred, is available atstep 227. For the case where the limit value for the amplitude was neverreached, the compensation is trivially just the amplitude (or, moreaccurately, parameter value) at T=t_(N), since n/N=1. Once the exposureis finished at 223, the process is then ready to return to the beginningand the pixel can be reset for the next photo or frame of video.

As noted, FIG. 4 represents the process flow for just a single pixellocation or row of pixels, where the read of the whole array is given inmore detail above. In the above embodiment, the time between each read(time given over for the loop) is based on the time to reset or read allthe rows in the array and return to the pixel. (In the embodiment below,it is based on the time to reset or read the groups into which the rowsare divided.) Note that in the above arrangement, all pixels have equalexposure intervals or sub-intervals—or, more accurately, essentiallyequal exposure intervals or sub-intervals due to the variations inherentin the design and operation of such devices.

The embodiment of the present invention described in the precedingparagraphs operates the CMOS sensor at a speed which increases inproportion to the number of interrogation cycles used. For theembodiment implementation example disclosed, this means that the sensorwould be readout 10 times faster to achieve the same effective exposuretime. In absolute terms, assuming a photo sensor with 9 million pixelsarranged in a grid of 3000 rows by 3000 pixels per row, the stated resetand interrogation cycle times of 3 milliseconds, and the stated use of10 interrogation cycles, an implementation employing an exposureinterval of 30 milliseconds would require the interrogation or reset of3000 rows of image sensor pixels in 3 milliseconds, and therefore thedata from each pixel sensor in each row to be sequentially clocked outof the sensor at a data rate of 1 MHz. This is relatively fast bycurrent standards, which would employ 1 interrogation cycle for a 30millisecond exposure interval, require the interrogation or reset of3000 rows of image sensor pixels in 30 milliseconds, and therefore thedata from each pixel sensor in each row to be sequentially clocked outof the sensor at a data rate of 100 KHz.

There are a number of ways of mitigating the need for increased sensorspeed. First, fewer than 10 interrogation cycle times can be used. Thiswill reduce the dynamic range increase from a factor of 10 down to afactor equal to the number of interrogation cycle times used. However,even a 2 times increase in dynamic range is a significant advance overthe dynamic range offered by the typical digital camera, and consumerswould welcome such an improvement.

Second, consider now an image sensor that has the incorporatedcapability for reset or readout of several groups of rows at a time,each at different locations on the image sensor. In this case, imageinformation may be effectively and simultaneously obtained from bands ofsensor rows for multiple interrogations cycles. In this embodiment,Group A, the rows of lines between “A” and “B” of FIG. 2, are reset andread out in accordance with the series of steps defined under theembodiment described earlier. If, for this example, a sensor is usedthat has 3000 rows of pixels, the A, B, C, D, and E lines groups wouldeach contain 600 rows, assuming an equal number of lines in each rowgroup. Note that equal row groupings are not depicted in FIG. 2. Readoutcircuitry 12 of FIG. 2 is responsible for converting these rows of pixelinformation and sending out image pixel data on the connection 14 asparallel data. This effectively cuts the data rate on each output lineby a factor of 5. For this example that means that the output data ratefrom the image sensor is 200 KHz, a number only a factor of two awayfrom the 100 KHz rate encountered when only one interrogation cycle isemployed, as is the case for prior art sensors and digital camerasystems. The 5 lines of parallel picture data emanating from the imagesensor would each be converted into digital form using separate A to Dconverters. The digital camera processor would accept these 5 digitaldata streams and using the approach previously outlined, process thispixel information back into a single image with extended dynamic rangefor each pixel.

As noted above, since the number of photons incident on bright areaswill accumulate more rapidly, the pixels from the bright parts of theimage use an effective shorter exposure time than the darker parts ofthe image. In addition to expanding the dynamic range of the sensorarray, this use of multiple, variable exposure times within a singleframe allows the bright areas of the image to be sharper, displayingless motion artifacts than the dim areas of the image. As the brighterportions of the image are often the more important parts, the resultantcrisper image in bright areas, while allowing darker areas longereffective exposure time, is another benefit provided by the presentinvention.

In alternate embodiments of the present invention, the use of thevariable exposure times within a single image can be user determined,rather than automatically based on the amount of light incident on agiven pixel. For example, a user could select a portion of the frame fora shorter exposure time in order to reduce motion artifacts or produceother effects within the camera. According to the embodiment, thisportion of the image could be selected beforehand or afterwards. Ifperformed afterwards, this can be implemented by overwriting the pixelsof the selected region with earlier recorded values.

Although in the main embodiments of the present invention a primaryaspect is the ability to expand dynamic while only storing a singlevalue for each sensor, the ability to concurrently store one or moreadditional values may also be incorporated. This would allow the sort ofoverwriting by an early value discussed in the last paragraph. Theseextra values could then be used for this sort of image sharpening aswell as for other selected effects, allowing for a sort of “digitaldarkroom” set of manipulations to be performed on the camera or imagingsystem. For example, in FIG. 5 the whole of the image is indicated at301 as displayed on, say, the viewer of the camera. Within this image, aportion 303 can then be selected for this sort of post-processingthrough the user interface. In addition to the sort of manipulationsalready described, the selected portion could also have extracompensation to brighten or increase contrast in low light regions or to“fix” a pixel that is near saturation, or just over saturation, whose“saturated” value was recorded, and a previous value would look betterin the final image.

FIG. 6 is a block diagram showing an example of wide dynamic range imagecapture system utilizing the above image sensor design as part of acamera. Although the image capture system is often described here as astill or video camera, more generally it can be a handheld device, suchas a phone or PDA, or other digital appliance that includes an imagecapture system. An image sensor 20 with multiple row readoutcapabilities is used together with an image processor 23 and memory 26.As the image is scanned, the processor receives the various pixel rowdata values and either stores them in appropriate locations in thememory 26 in a manner convenient for the later mathematicalmanipulations, or performs the dynamic range extension processing asoutlined in Step (10) above, “on the fly”, in other words, as the imagedata is received. Image scan control is done with the control signals22, and data to the processor is on wire 21, which for this examplerepresents multiple buses of sensor data after A to D conversion, inorder, as previously explained, to reduce data rate requirements. Ifsensor speeds and processor capabilities allow, line 21 can be a singleline, and the above FIG. 6 would represent an exemplary apparatus forboth the first and second embodiments given above. For somearchitectures, the image data may connect directly to the memory 26.Data in the memory is located using the address and control lines 24,and transferred between the processor and the memory on lines 25. IfSteps (1) through (9) above are followed, the processor scans the storeddata and produces image data output incorporating the best image datafor each pixel location. In this way the dynamic range of the image isextended by the ratio of the longest effective exposure time to theshortest effective exposure time read out.

The image processor outputs its final image data on the wires 27 to theremainder of the imaging system. This data may be the pixel data asgenerated by the mathematical algorithms driving processor 23, or otherinformation derived from the image data.

In addition to these elements, the camera or other device will includethe various other elements as are familiar in the art. These wouldtypically include an optical system 30, containing the lenses andaperture that supply the illumination to expose the image sensor, and adisplay unit or storage 32, such as a memory card, to store the images.The various other elements and control circuitry (exposure control andso on) are not explicitly shown in the figure.

As noted in the Background section, there are various prior art methodsand apparatus that have been proposed to increase the dynamic range ofphoto sensors in general, and CMOS sensors in particular. However, manyof these devices suffer from the need to either use non-standard CMOSsensors that are significantly more costly to manufacture and controlthan the CMOS sensors commonly in use, or require significantly moreimage buffer memory to properly operate. One proposed approach forincreasing CMOS sensor dynamic range, while using standard CMOS sensors,is to capture two to more images, taken very close together in time, atdifferent exposures, at least one which displays the shadow detail ofthe image (taken at a high exposure) and at least one which displays thehighlight detail (taken at a low exposure). These multiple images, whichneed to be corrected for any motion present in the images occurringbetween individual image capture, are combined into a single image thatdisplays both shadow and highlight detail. This can provide someenhancement of dynamic range; however, the processing complexity tomotion compensate and appropriately combine the two or more capturedimages, and the additional buffer memory required, is excessive bytoday's standards, and therefore this solution, to date, has not beenshown as a practical answer to the problem. As the present inventiondoes not require extra buffer space to achieve CMOS sensor dynamic rangeextension, and minimal additional processing is required, it is believedto be a better way to solve the problem. By using only a singleexposure, without resetting during the exposure interval, and storingand reading out only a single value per pixel, the present inventionresolves the imaging system manufacturing cost and control issues, andprovides a practical solution that in the past has not been available.

The present examples are to be considered as illustrative and notrestrictive, and the invention is not to be limited to the details givenherein, but may be modified within the scope of the appended claims.

1. A method for increasing the dynamic range of an imaging device, themethod comprising the acts of: exposing an image sensor of the imagingdevice for a continuous exposure period, the image sensor including apixel array; determining an output level of a pixel of the pixel arrayat a first time interval within the exposure period; storing the outputlevel of the pixel at the first time interval as a pixel output valuefor said pixel; comparing the pixel output value to a limit value;compensating the pixel output value based, at least in part, on saidcomparing when the pixel output value exceeds the limit value; andstoring a compensated pixel output value as the pixel output value basedon said compensating, wherein the compensated pixel output value isbelow the limit value.
 2. The method of claim 1, wherein the continuousexposure period relates to a period for capturing image data for a frameby the image sensor.
 3. The method of claim 1, wherein compensating thepixel output value includes scaling the pixel output value to a valuebelow the limit value.
 4. The method of claim 1, wherein compensatingthe pixel output value is based on detection of a second pixel outputlevel associated with a second time interval within the exposure period,wherein the second time interval is prior to the first time interval. 5.The method of claim 4, wherein the compensated pixel output value isscaled to a value above the second pixel output level.
 6. The method ofclaim 1, further comprising determining output levels for each pixel ofthe pixel array for the first time interval, and compensating aplurality of pixel output values at the first time interval for aplurality of pixels of the pixel array.
 7. The method of claim 1,wherein compensating the pixel output value is based on at least one ofa user specified pixel limit and user specified exposure period.
 8. Themethod of claim 1, further comprising determining an output level of thepixel at a plurality of intervals within the exposure period, andcomparing the output level of the pixel for each of the plurality ofintervals to the limit value.
 9. The method of claim 8, wherein theplurality of intervals within the exposure period relate to unequal timeintervals.
 10. The method of claim 1, wherein compensating the pixeloutput value is based on a relative duration of a subinterval of thecontinuous exposure period, wherein the relative duration of thesubinterval is determined based on the ratio of a number of outputlevels determined below the limit value to the total exposure period.11. The method of claim 1, further comprising storing compensated pixeldata for a plurality of pixels of the pixel array to generate an imagewith increased dynamic range.
 12. An imaging device comprising: an imagesensor including a pixel array, the image sensor configured to exposethe pixel array for a continuous exposure period; a memory; and aprocessor coupled to the memory and the image sensor, wherein processoris configured to: determine an output level of a pixel of the pixelarray at a first time interval within the exposure period; instruct thememory to store the output level of the pixel at the first time intervalas a pixel output value for said pixel; compare the pixel output valueto a limit value; compensate the pixel output value based, at least inpart, on comparing the pixel output value to the limit value when thepixel output value exceeds the limit value; and instruct the memory tostore a compensated pixel output value as the pixel output value basedon the pixel output value compensation, wherein the compensated pixeloutput value is below the limit value.
 13. The device of claim 12,wherein the continuous exposure period relates to a period for capturingimage data for a frame by the image sensor.
 14. The device of claim 12,wherein compensating the pixel output value by the processor includesscaling the pixel output value to a value below the limit value.
 15. Thedevice of claim 12, wherein the processor is further configured tocompensate the pixel output value is based on detection of a secondpixel output level associated with a second time interval within theexposure period, wherein the second time interval is prior to the firsttime interval.
 16. The device of claim 15, wherein the compensated pixeloutput value is scaled to a value above the second pixel output level.17. The device of claim 12, wherein the processor is further configuredto determine output levels for each pixel of the pixel array for thefirst time interval, and compensating a plurality of pixel output valuesat the first time interval for a plurality of pixels of the pixel array.18. The device of claim 12, wherein compensating the pixel output valueby the processor is based on at least one of a user specified pixellimit and user specified exposure period.
 19. The device of claim 12,wherein the processor is further configured to determine an output levelof the pixel at a plurality of intervals within the exposure period, andcomparing the output level of the pixel for each of the plurality ofintervals to the limit value.
 20. The device of claim 19, wherein theplurality of intervals within the exposure period relate to unequal timeintervals.
 21. The device of claim 12, wherein compensating the pixeloutput value by the processor is based on a relative duration of asubinterval of the continuous exposure period, wherein the relativeduration of the subinterval is determined based on a ratio of the numberof output levels determined below the limit value to the total exposureperiod.
 22. The device of claim 12, wherein the processor is furtherconfigured to store compensated pixel data for a plurality of pixels ofthe pixel array to generate a image with increased dynamic range.
 23. Amethod for increasing the dynamic range of an imaging device, the methodcomprising the acts of: exposing an image sensor of the imaging devicefor a continuous exposure period, the image sensor including a pixelarray; determining an output level of a pixel of the pixel array at afirst time interval within the exposure period; storing the output levelof the pixel for the first time interval as a pixel output value forsaid pixel; comparing the pixel output value to a limit value;determining an output level of the pixel at a second time intervalwithin the exposure period when the pixel output value does not exceedthe limit value; storing the output level of the pixel for the secondtime interval as the pixel output value for said pixel; comparing thepixel output level associated with the second time interval to the limitvalue; compensating the pixel output value associated with the secondtime interval when the pixel output value associated with the secondtime interval exceeds the limit value; and storing a compensated pixeloutput value as the pixel output value based on said compensating,wherein the compensated pixel output value is below the limit value. 24.The method of claim 23, wherein the continuous exposure period relatesto a period for capturing image data for a frame by the image sensor.25. The method of claim 23, wherein compensating the pixel output valueincludes scaling the pixel output value to a value below the limitvalue.
 26. The method of claim 23, wherein the first time interval isprior to the second time interval.
 27. The method of claim 26, whereinthe compensated pixel output value is scaled to a value above the firstpixel output level.
 28. The method of claim 23, further comprisingdetermining output levels for each pixel of the pixel array for thefirst time interval and second time interval, and compensating aplurality of pixel output values for the pixel array.
 29. The method ofclaim 23, wherein compensating the pixel output value is based on atleast one of a user specified pixel limit and user specified exposureperiod.
 30. The method of claim 23, further comprising determining anoutput level of the pixel at a plurality of intervals within theexposure period, and comparing the output level of the pixel for each ofthe plurality of intervals to the limit value.
 31. The method of claim30, wherein the plurality of intervals within the exposure period relateto unequal time intervals.
 32. The method of claim 23, whereincompensating the pixel output value is based on a relative duration of asubinterval of the continuous exposure period, wherein the relativeduration of the subinterval is determined based on a ratio of the numberof output levels determined below the limit value to the total exposureperiod.
 33. The method of claim 23, further comprising storingcompensated pixel data for a plurality of pixels of the pixel array togenerate a image with increased dynamic range.